POOL-DEVICE-KEY: one scratch pool per DEVICE, rebased off a stale base, with the dgx pair (#516, #486, #614) - #617

…rough DBuf

The gate for #516, and it is RED at this commit deliberately: the fix lands in
the next one, so git itself records that the test was seen failing rather than
written to fit a change that had already been made.

    test cases:  4 |  2 passed | 2 failed | 0 skipped
    assertions: 15 | 10 passed |  5 failed
    Status: FAILURE!   exit=1

The decisive assertions are not the pointer comparison but the ownership ones:
`b.Owns(on_b)` is FALSE and `a.Owns(on_b)` is TRUE -- device 1 did not merely
receive an equal pointer, it received a block that device 0's allocator made.

It allocates through `dense_attn::DBuf`, the seam every production forward draws
scratch from, so it holds the path the LTX-2.5 device suite crashes on rather
than a paraphrase of it. No GPU, no checkpoint, no NAS, milliseconds: two fake
backends stand in for two devices, the technique test_backend_multidevice and
test_reference_tier already use.

Two of the four cases pass now and must KEEP passing. Reuse on one device
returns the identical block, and two byte sizes in one class still share a
block. Without them a "fix" would be indistinguishable from VT_POOL_BYPASS=1,
which also separates the devices -- by reinstating the per-op cudaMalloc/cudaFree
sync storm the pool exists to remove.

Every case uses its OWN size class, so no case can be decided by what another
left in a free list, including under --order-by=rand. Two test-side properties
are load-bearing: a fake backend never returns a block to the C allocator (these
cases compare pointer identity ACROSS a free), and no fake backend is destroyed
before exit (the pool is keyed on backend identity, and a reused address would
let one case's pool answer another case's question).

Refs #516

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]

…l "the pool"

Turns the RED-first gate green: 8 cases / 26 assertions SUCCESS, from 2 of 4
cases failing at f4be8a4e2.

The device is now STRUCTURAL, not a field a caller has to remember. A
`DevicePool` is bound to one backend at construction, `Pool(b)` resolves the pool
for a device, and the no-argument `Pool()`/`AuxPool()` are GONE -- "the pool"
without a device was the defect, so it is no longer expressible. `vt::Backend*`
is the device identity, since the registry hands out exactly one Backend* per
Device{type,index}; that puts the device in the key with NO new virtual on
vt::Backend, so not one backend implementation is touched.

Lookup sits on the hottest allocation path in the tree -- a DBuf resolves its
pool on every construction -- so a thread-local last-(backend,pool) memo makes
the steady state a single pointer compare. No hash, no lock, on the hit path.

Every pool operation VERIFIES its backend and throws. Deliberately not an
`assert`: the gate builds are Release/NDEBUG, where an assert compiles out and
the silent cross-device hand-off returns. The only way to reach the throw is an
`ActivePoolScope` aimed at another device's pool, which is exactly the mistake
this row makes impossible to make quietly.

`DBuf::ReleaseShared()` replaces 31 copy-pasted shared_ptr deleters that closed
over a byte count ALONE. Those named neither the device nor the pool, so they
returned another device's block -- and, separately, an AUX-STREAM block -- to the
main pool. That second bug was live on every path that used the idiom. Each site
goes from three lines to one, and the backend-less `Put` overload is removed with
its last caller.

Two more instances of the same ambient-device assumption, found while fixing it
and repaired here rather than left to be rediscovered: the decode-graph
`PersistentDecodeInputPool` was a process-wide static, and `ResolveDevicePool
Policy` memoized whichever device asked FIRST and applied its residency cap to
every later one. Both are now per device. Byte-neutral today (every platform's
`device_pool_cap_bytes` is 0), which is why it is safe to do here.

The two per-caller workarounds for this bug are REMOVED, not kept: the
`ActivePoolScope` around the LTX-2.5 bf16 CPU arm (the only test in the tree that
reaches the SILENT direction) and the per-arm pools in the DeepSeek-V2 CUDA-vs-CPU
case. Both are again detectors instead of callers that were scoped away from the
hazard, and each carries a comment saying not to re-add the scope: it would pass
whether or not the pool is correct.

Mutation-proven, each restored byte-for-byte afterwards. Drop the device check ->
only the refusal case fails (7/8). Also collapse the pool table to one pool ->
4 of 8 fail, exactly the two direction cases plus Drain plus the refusal. Make
ReleaseShared use the device's main pool instead of the buffer's own -> only the
scoped-pool case fails. The two "still a pool" cases (reuse returns the identical
block; two sizes in one class share a block) pass throughout, so this is not
VT_POOL_BYPASS wearing a fix's clothes.

NO docs/FEATURES.md OR docs/USAGE.md UPDATE, AND HERE IS THE ARGUMENT FOR IT,
attached to the diff it excuses because this protocol has no waiver registry.
`check-doc-checkpoint.py` classifies any edit under `src/vllm/model_executor/
models/` as `feature_surface` and any edit under `include/vllm/` as
`user_usage`, so it asks this commit for both. Nothing here is either. No
feature, model, backend or quantization surface changes; no command, C API,
config key, install step or workflow changes. The 24 model TUs are touched by a
mechanical three-lines-to-one call-site rewrite, and the header change is an
internal allocator seam that `include/vllm.h` does not expose. Writing filler
into two public projections to satisfy a path prefix would make them less true,
not more. AGENTS.md states the same rule in prose the other way -- "Editing
src/, include/, or tests/ on its own owes none of these" -- and says the checker
and the prose are deliberately not kept in sync; this is one of the places they
disagree. The gate also already fails on this branch's base for the same reason
at b0aa475 and d67f812, so this is not a new red. A reviewer who does not
accept the argument should not merge it.

Refs #516

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]

…, and what is still unattributed

What neither the code nor git records: what was measured, what was rejected, and
why each default is set the way it is.

Both symptoms were reproduced at the RED commit on GB10 and both are gone at the
fix. The SIGSEGV direction: `--rand-seed=7` exit 139 with the trap it carries --
44 assertions, 0 failed, beside a crash. The SILENT direction: the shipped 21B
FP8 DiT returning `REQUIRE(std::isfinite(v))` FAILED, which needed the opt-in
fixture actually satisfied, so the checkpoint was proven READABLE first -- with
the NAS down that case SKIPS and the suite reports SUCCESS, an environmental
failure wearing the shape of a repair. After: 13/13 and 6176 assertions in every
ordering, with no per-case pool scoping left anywhere in the file.

#486 was a hypothesis and is now a measurement, in both directions: at the RED
commit `test_minimax_h3` SIGSEGVs with the pool on and is 79/79 under
`VT_POOL_BYPASS=1`; at the fix it is 79/79 with the pool ON.

The nine dgx full-suite failures are recorded as UNATTRIBUTED rather than
explained away. The dgx BEFORE arm could not be run -- the box was at 99-100%
disk, so two 31 GB trees would not fit -- and two bounded `flock -w 2700` waits
for the shared GPU lock expired without acquiring. One of the nine IS resolved:
`test_capi` reproduced on the CPU host, where this change cannot cause it, and is
8-of-8 green standalone with per-run times spanning 0.78 to 339.88 s. The other
eight get a named next step and the script to run it, not an adjective.

Also records the ENOSPC re-verification the operator asked for (no build log
contains `No space left`; local gates re-run chained to their build, ninja "no
work to do" first, so nothing here is a stale binary), the measured blast radius
(42 of 402 binaries instantiate a DevicePool; exactly one instantiates more than
one on a single-backend host), the rejected designs, and two things left open on
purpose: why a host block yields a uniform quiet NaN on GB10 rather than running
correct-but-slow, and `MoeAuxStreamFor` keying on device INDEX alone -- the same
family, unreachable today because its only call site is gated on
`SupportsAuxStream()`.

Refs #516

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]

…ew findings (#516)

A fresh review returned FAIL on the LANDING STATE while endorsing the fix
itself. This closes all six findings. The code the reviewer endorsed --
the device key, `ReleaseShared`, both workaround removals and
`test_device_pool` -- is unchanged in behaviour.

F1 (HIGH). The branch was based on `row/MODEL-DIFFUSION-LTX25` @ `aac24761`,
which is not an ancestor of `origin/main`, and `git merge-tree` CONFLICTED in
`device_pool.h`. Since that merge base, main changed the same file in
`49539559d` / `8fa2ecdbb` (Windows contracts, #117) with three changes the
reviewed header lacked: `__builtin_clzll` -> `std::bit_width`, an
`std::overflow_error` guard in `ClassOf`, and `static SizeClassForTest`. A
resolver taking the row's side -- the heavily rewritten side -- would have
silently reverted the portability fix and the overflow guard, and
`origin/main:tests/vt/test_cpu_isa_x86.cpp` calls `SizeClassForTest` NINE
times including `CHECK_THROWS_AS(..., std::overflow_error)`, so it would not
have compiled. The four commits are rebased onto `row/MODEL-DIFFUSION-LTX25`
@ `2d437d5a9`, which now contains main, and `device_pool.h` was resolved BY
HAND: all three of main's changes sit ON TOP of the device-keyed rewrite, not
instead of it. `git diff row/POOL-DEVICE-KEY..HEAD -- device_pool.h` is exactly
those three hunks and nothing else. `test_cpu_isa_x86` is 6 cases / 8242
assertions / SUCCESS.

The row keeps its base. `tests/vllm/models/test_ltx2_device.cpp` does not exist
on `main` at all and it is the only test exposing the SILENT-NaN direction, so
deleting its workaround -- which the row requires, because a list of remembered
callers is what this fault was -- needs LTX-2.5 to land first.

F2 (MEDIUM). The row had no record surface, and merging would have made an
existing record FALSE. `#516` is now in the issue table of
`.agents/roadmap_v1.md`; the spec carries a `## Now`; `docs/STATUS.md` carries
the paragraph under "Backend detail"; and `porting-inventory.md` §L8, which
still said the shared `DevicePool` "is DEVICE-BLIND ... repairing it is owed as
its own row", now says what actually happened. `docs/BENCHMARKS.md` is
deliberately not written: this row claims no measurement on any axis.

F3 (MEDIUM). Recorded in the spec's §11 with the dgx BEFORE/AFTER pair, the
disk and lock state at both ends, and what remains unattributed.

F4 (LOW) is CORRECT and the assertions are reworded. Enumerated at the base
commit: none of the nine `Release()` sites (`gemma4_moe.cpp:1197,1541`,
`qwen3_5.cpp:6324,6520,6820,7100,7134,8034,8065`) is inside or under any of the
four `ActivePoolScope` regions (`laguna.cpp:2574`, `qwen3_5.cpp:5468,8644,8964`),
which are leaf-ward of all of them. The aux-pool half was LATENT, not live, and
`device_pool.h`, `dense_device_glue.h` and spec §4 D4 now say so. `ReleaseShared`
stands on its own merits and is untouched.

F5 (LOW). Both debug lanes are GREEN, which matters because §10 hands
`VT_POOL_BYPASS=1` to the next reader as the cheap discriminator: a suite that
reds under the lane it recommends costs that reader an hour deciding whose red
it is. A case whose subject is REUSE now states what the ACTIVE lane does, and
the size-class case states sharing by default and SEPARATION under
`VT_POOL_EXACT` -- which is spec §5 T1.3's second clause, promised since the
spec was written and asserted nowhere until now.

F6 (LOW). The per-device-type memoization made `platforms::GetPlatform` a
per-type call, so an unregistered platform now throws where it used to inherit
the first device's cap. Correct, and it had no test; it has one. The
`cached[...]` index gains the bound `platforms::Index()` already applies to the
same value, in both mirrored copies.

Nit: spec §5 T1 said the fakes sit on the `kXPU` slots; the test uses `kCPU`
indices 0/1 and is right, because `kXPU` has no registered platform and every
pool case would have measured the platform registry instead. The spec is
corrected and `kXPU` now earns exactly the one case that is about that throw.

New mutation evidence, on this tree:
  * bound-check + refusal removed (`ResolveDevicePoolPolicy` defaults an
    unregistered platform to cap 0) -> 8 passed / 1 failed, the T1.7 case only
  * `VT_POOL_EXACT` made inert in `ClassOf` -> 8 passed / 1 failed under
    `VT_POOL_EXACT=1`, 4 assertions, the size-class case only
  * `Bypass()` forced false -> 4 passed / 5 failed under `VT_POOL_BYPASS=1`
Tree restored byte-for-byte after each (md5 verified).

CPU host, RESOLVED tree, CLEAN rebuild: `BUILD_EXIT=0`, zero warnings under
`-Werror`, zero `No space left`/`BFD assertion`, `ctest -N` 412, full `ctest`
412/412 passed exit 0. `test_device_pool` 9 cases / 30 assertions SUCCESS
(24 under bypass, 31 under exact, all SUCCESS). Every §6 baseline is where it
was except `test_ltx2_vae` 16/1816 -> 33/2602, which is `6c9374ebc` on the
CAMPAIGN branch (the VAE encoders) and not this row -- this row does not touch
that file. That drift is exactly what F1 predicted a stale base would hide.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]

…ures attributed (#516, #486, #614)

Spec §11. The review's F3 said this row "must not land on a CUDA-unattributed
gate": every direct symptom of #516 needs a GPU, both removed workarounds guard
CUDA cases a CPU host never executes, and §10 had nine dgx failures with NO
BEFORE arm because the box was at 99-100% disk with a single 31 GB build tree.

The pair now exists: both arms clean-built from the SAME base, back to back on
one box, `ctest -j 1`, all three MANDATORY confirmations
printed for each and zero `No space left` in four logs.

  BEFORE  aa6aa0e, no fix   10 failed of 449   lock 08:26:13Z-09:22:00Z
  AFTER   this branch          9 failed of 450   lock 09:40:20Z-10:34:02Z
                                                 (13m45s bounded wait, not forced)

THE DENOMINATORS DIFFER BY ONE BECAUSE THIS ROW ADDS A TEST. 449 -> 450 is
test_device_pool existing in the AFTER arm and not the BEFORE one; it is not
drift, and 10-of-449 read against 9-of-450 without that is off by one test. On
the 449 both arms share: 10 failures before, 9 after, and exactly ONE leaves.

  test_minimax_h3    ***Exception: SegFault 11.73s  ->  Passed 19.06s
  test_device_pool   (does not exist)              ->  Passed

The other nine are the SAME set in both arms, and each is now matched to an
issue rather than left as a name: #233 (test_serve_low_tools, test_linear_method,
test_glm4_moe_lite_paged_engine), #248 (test_capi, test_qwen3_apc_e2e,
test_minicpm3_paged_engine, test_llama_paged_engine), #305 TENTATIVELY
(test_ops_gdn), and #614, filed by this row, for test_internlm2_paged_engine,
which had no issue anywhere. So all nine are MEASURED pre-existing rather than
argued. This row does not fix them and does not adopt them.

test_capi IS NOT THE TIMING FLAKE §10 CALLED IT, and the correction matters.
§10's flake evidence -- 8/8 under --repeat until-fail, wall times 0.78s to
339.88s -- came from the CPU HOST, where the test fails without crashing. On dgx
it SEGFAULTs, in BOTH arms, in 1.95s and 1.97s, under ctest -j 1. A sub-2-second
segfault is not a three-orders-of-magnitude timing spread, and #248 says so in as
many words. Pre-existing either way, but it goes back to #248 as a crash rather
than into a flake story it does not fit. Whether the two arms' crashes are the
SAME crash is NOT established: neither LastTest.log survives (both 34 GB trees
were deleted to keep a 98%-full box safe) and these runs were plain ctest -j 1
with no --output-on-failure, so the recorded evidence is the signal, the index
and the duration.

It also re-confirms #486 in BOTH directions on ONE box, 14 minutes apart, same
binary recipe, same lock: SEGFAULT with the pool device-blind, Passed with it
device-keyed. §10 asserted that from two separate sessions; it is now one paired
measurement.

Both 34 GB build trees were deleted after their arm. The box was at 98-99%
throughout and ended at 94 GB free.

The pair was taken at base aa6aa0e; the branch is now rebased onto 310fa16,
fifteen commits later, and the CPU gate was re-run in full there (415/415, clean
rebuild, zero warnings). Chasing a moving campaign branch with a five-hour paired
CUDA gate does not terminate, so §11 names the base the pair was taken at and
states what would invalidate it: none of the fifteen intervening commits touches
device_pool.h, dense_device_glue.h, the DBuf deleters or any pool accessor, and a
tree-wide sweep for a device-less Pool()/AuxPool()/ActivePool() comes back empty
on the new base.

§11 also records what §10 must no longer be read as saying: every number in it
was measured on a tree missing main's last 201 commits, and none of it is
carried forward. `## Now` is updated to match -- the nine are closed against
this row, and the two things that remain unestablished (the ATS/quiet-NaN
mechanism, and `MoeAuxStreamFor`'s index-only key) are named rather than
quietly dropped.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]